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Abstract: This study investigates the relationship between the designated research project 
system independent variables of Labor, Travel, Equipment, and Contract total annual costs 
and the dependent variables of both the associated matching research project total annual 
academic publication output and thesis/dissertation number output. The Mahalanobis 
Taguchi System (MTS) pattern recognition methodology was utilized in the three-year, 
3000+ research project case study to identify which research project system variables are 
responsible in both magnitude and degree for the associated publication & thesis and 
dissertation research project outputs. The selection of the MTS “abnormal” and “normal” 
data set boundary was specifically chosen in an attempt to define a “successful” from 
“unsuccessful” research project metric at the research unit, tenure track versus non-tenure 
track, and individual research project principal investigator organizational levels. The 
findings of the study are directly compared against the associated level budget percentages 
changes over the same time period. Through such concrete research system identification, 
research administrative personnel have the possibility ofdirectly identifying both the research 
project system “imp actors”as well as the association between likely effects of reduced or affected 
research system impacts on the research outputs themselves. 
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Introduction 

Modern university research projects, even though they are still primarily leadership driven by 
individual Principal Investigator (PI)/Program Director (PD)-equivalent faculty members, 
must today utilize and rely on a series of institutional infrastructure systems for their facilities, 
instrumentation, travel, contracting, labor, and administrative needs to function effectively. Each 
of these systems has various costs associated with them (Haley, 2009; Haley 2011; Grieb, Horon, 
Wong, Durkin, & Kunkel, 2014). 

This paper addresses the use of systems engineering-based management concepts to respond to 
and develop an advanced research systems administrative and management information approach 
prototype in order to allow research universities to improve the effectiveness of their research 
infrastructure administrative tools and policies. 
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The International Organization of Standards (ISO) Subcommittee on Software Engineering 
(SC/7) developed the current measurement standard for software measurement processes. “This 
International Standard identifies the activities and tasks that are necessary to successfully identify, 
define, select, apply and improve measurement within an overall project or organizational 
measurement structure” (ISO, 2007). However, while this standards title (referred to as ISO/ 
IEC 15939) uses the terminology of software engineering, it is explicitly meant to also refer to 
systems engineering (Frenz, Roedler, Gantzer, & Baxter, 2010). ISO/IEC 15939 is also defined in 
terms of fields of application. In the context of a university research administrative organization, 
one of the fields of application, “by a supplier to implement a measurement process to address 
specific project or organizational information requirements” can be seen as representing the 
necessary measurement needs of a research university administrative component organization 
(ISO/IEC 15939,2007). 

Ofnote, ISO/IEC 15939 is not alibraryofmeasurements nor does it provide any recommendation 
on which measures apply to an individual project or organization. It merely defines a process 
supporting the construction of defined and tailored measures for an organization’s individual 
requirements. 

From an organizational management perspective, ISO/IEC 15939 also details the steps necessary 
for an organization to ensure that their measurement processes are optimized to form a set of 
“requirements” to ensure the maximum utilization of these same measurement processes. Figure 
1 depicts this process, including the four fundamental measurement task activities. 


Measurement Process Model 



PSM VfRStON e 2 * 3 

Figure 1. ISO/IEC 15939 Measurement Process Model. (From ISO/IEC 15939, 2007) 
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Institutional Background 

Between 2009 and 2012, the Naval Post Graduate School (NPS) experienced an unprecedented 
growth rate exceeding thirty percent annually in its research funding. From ~$75M in 2009 
to over ~$150M in 2012; the bredth and scope of research work done at NPS increased 
dramatically (NPS, 2014). Such growth was not without its problems, however. Several major 
new research program investments failed to achieve their intended results and goals. A root cause 
analysis was conducted at the project level by the Chairs and Dean of the associated schools. 
Additionally, several internal and external reviews were conducted of both institutional research 
project acceptance and research project review processes. As a consequence, the top level research 
management and administrative level organization at NPS, the Research and Sponsored Programs 
Office (RSPO), began exploring possible additional technical and management processes in 
order to assist in preventing such similar occurrences from occurring again. 

As a result of these technical and management process actions, the RSPO has adopted a multi¬ 
pronged institutional approach to managing the acquisition of key information required for 
support, improvement, and strategic planning for critical research activities. These include: (1) 
the development, formation, and maintenance of an institutional work acceptance policy and 
joint academic, research, and leadership committee to monitor and serve as a senior decision 
body for research program acceptance according to the institutional strategic plan; and (2) the 
development, formation, and maintenance of an evolving research system, staff, and project 
research output measurement information construct. This latter element will form the major 
portion of this paper. 

Research Administration Background 

Education, research, and services constitute the three major responsibilities of universities (Boyer, 
1996). Because of the inherent complexity of optimizing and simultaneously balancing these 
three different mission areas, the resultant university structure that results from it is one that 
requires a critical, evolving, and well thought out management process. This would include the 
specific research management structure employed (Bosch & Taylor, 2011; Pettigrew, Lee, Meek, 
& Barros, 2013). Furthermore, in selecting the elements necessary in formulating the elements of 
this research management structure, Mintzberg (1979) lists four generic parameters, including an 
information based decision making system as being necessary (Haines, 2012). 

With respect to this decision making system Kirkland (2008) notes “a system to identify any 
emerging problems at an early stage” as being critical. Taylor (2006) also supports this concept by 
stating that research management administration should be seen as “encouraging, supporting and 
monitoring” project entities. 

The general advantages of developing a systematic information based measurements tool basis 
as a tool for research administration are not new. Haines (2012) has pointed out that their 
use includes establishing and oversight of research business processes, defining responsibilities, 
controlling expectations, driving team motivations, assessing research staff performance, as well 
as upgrading tools for both research decision making and prioritization. 
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While many research administrative organizations use various information measurement 
constructs as a key portion of their overall responsibility, as Nguyen, Huong, and Meek (2015) 
point out, “the need for an effective, evidence-based metric standard that captures the complexity 
of the (Research Management) field remains unmet.” 

As Nguyen et al. (2015) have also pointed out, universities typically use an amalgam ofpublication 
information, peer review, or a combination of the two former to quantify research personnel 
outputs at the individual principal investigator, department, or school levels. This combination, or 
bibliometrics, includes such elements as impact factors and/or citation rates. While there are both 
pro arguments (Taylor, 2011) and contra arguments (Adams, 2009) to the use of bibliometrics, 
the inclusion of specific system derived information directly relevant to the integrated System 
of System (SoS) research project outputs has, to the best of the authors’ knowledge, not been 
attempted for inclusion. 

Literature Review 

Cost functions as almost a universal sanctioned means of exchange and value throughout 
communities (Newlyn, 1978). Investments in a particular quantity, whether it be equipment, 
labor, project, etc., can all be associated with the appropriate process if there is an accurate 
accounting system to assign “cost” in dollars to these processes and interactions (Langford, 2012). 
The system engineering usage of cost constituting the basis for modelling is also well known 
(Boehm et al., 2000; Blanchard, 1998). 

Any performance management information system which has the capability to actively predict 
performance, must also meet two additional criteria: (1) it must have a program management 
framework; and (2) it must also have a procedural framework (Folan & Browne, 2005). The 
numeric basis of cost makes it attractive as many other gauges of systems measurement (Beamon, 
1998). Beamon (1998) also concluded, however, that it is improbable that a unitary performance 
measure like cost will be sufficient, hence a conglomerate of performance measures is demanded 
for precise assessment. 

The need to further classify research university project performance measures as they relate to 
performance information management is apparent from the fact that effective performance 
information management involves more than merely quantifying usefulness or benefit as the 
produced outcome of any organizational undertaking (Macbryde & Mendibil, 2003). Monitoring 
the processes responsible for those outcomes themselves is equally as important in order to 
influence the determination of the usefulness or benefit (Busi & Bitici,2006). The outcome-process 
difference is also related to a ‘systems’ view of research university organizational functioning, in 
that the effectiveness or performance of the organizational system in question will itself then be a 
nascent property of interaction and processes between which and within which all elements that 
comprise the system as a whole may function (Atkinson, Waterhouse, & Wells, 1997). 

Systems engineering (SE) involves the utilization of multiple academic disciplines to integrate 
various concepts “encompassing a wide range of engineering fields and associated analytical 
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thought processes” (KossiakofF & Sweet, 2003). It further defines an overall ‘systems’ thinking 
to consider the broader perspective that embraces a systems perspective as well as the interaction 
and integration of individual parts (Cowan, Allen, & Mistree, 2006; Haskins, 2006). Systems 
Engineering also bears a strong resemblance to an implementation of General Systems Theory 
in that the “general nature of a complex problem is to find a solution using systems ideas and 
principles” (von BertalanfFy, 1962). 

Gorod, Sauser, and Boardman (2008) suggest that there are eight measures (focus, boundaries, 
problem, structure, goals, approach, timeframe, and centricity) to compare system versus SoS 
engineering approaches. If the focus is on an integrated system, boundaries are dynamic in nature, 
problems tend to be of an emergent nature, structures of the organization are best described as a 
network, goals are pluralistic, timeframes are continuous, the nerve center of the organization is a 
network itself, then a SoS SE approach would be considered the better alternative (Gorod, Sauser, 
& Boardman, 2008). 

DeLaurentis and Callaway (2004) define a SoS as “the combination of a set of different systems 
[that] forms a larger system of systems that performs a function not performable by a single 
system alone.” In the context of a university research administration application, however, the 
definition by Jamshidi (2009) as “an integration of a finite number of constituent systems which 
are independent and operable, and which are networked together for a period of time to achieve 
a certain higher goal” seems to serve the present application best. 

To be considered as a SoS, there are at least five identified traits: “operational independence, 
managerial independence, emergent behavior, evolutionary development, and geographical 
distribution” (DeLaurentis, 2007; Boardman, DiMario, Sauser, & Verma, 2006). Maier (1998) 
interprets operational independence as: “ifthe SoS is disassembled into its component systems, the 
components systems must be able to usefully operate independently.” Managerial independence 
is defined similarly as: “the component systems not only can operate independently, they do 
operate independently.” 

In summary, a SoS executes both tasks and designs that cannot and are not either inhabiting any 
single constituent system (Engell, 2014). These functions are the nascent constituent elements 
of the total SoS itself and may not be confined to any single constitutive system. The foremost 
functions of the SoS emerge and are in turn satisfied by the SoS (Maier, 1998). Hence, SoS are 
comprised of multiple systems that are each managed and operated independently. At the SoS 
level, they deliver additional benefits. The enterprises at the systems level are all seen to have 
managerial and operational independence. Together, however, the individual component systems 
collaborate to develop and operate the SoS (Maier, 1998). 

The current state of NPS research administration operations is seen to be requiring a more 
developed research output measurement information construct. NPS is not alone in this regard 
(Nguyen et al., 2015). A systems engineering-based method of analysis offers us one possible way to 
gain additional perspective into the many complexities observed in managing a research university 
construct by providing an additional dimension through which to view research conduct. 
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Method 

At NPS, the increased fidelity of individual research project financial data has provided valuable 
spending pattern accounting information. It was not until 2011, however, that questions arose 
whether the individual total annual system cost structure items afforded by use of the institutions 
Kuali Financial System (KFS) (labor, equipment, etc.) may be treated as representative of the 
individual systems they originate from in order to possibly afford additional insight into NPS 
RSPO research operations (Kuali, 2014). This further implied that a possible new integrated 
RSPO planning and control information mechanism could be constructed, utilizing continuous 
process measurement techniques to track the state(s) of the system(s) variance trends. 

The detailing of research project total expenditures as broken down by labor, equipment, travel, 
contracting, and “indirect” costs is conceived as five operational and managerial independent 
systems functioning as a SoS. Labor, Equipment, Contracting, Indirect, and Travel are all 
individually operationally separate and managed organizational entities. This includes their 
“input personnel, hardware requirements, software requirements, facility spaces, office policies, 
and in house documents that interact substantially through processes, feedback, and boundaries” 
(INCOSE, 2006). The outputs from each system, with the exception of Indirect, include 
“specifically different qualities products, properties, characteristics, functions, behaviors, and 
performances” (INCOSE, 2006). The individual system elements (Labor, Equipment, Travel, 
and Contracting) are all “managerial and operationally independent elements” (Boardman et 
al., 2006). They also possess the traits of “emergent behavior and evolutionary development” 
(DeLaurentis, 2007). Geographical distribution of elements may be more problematic. The 
individual system offices are physically co-located within a l h square mile area, however, the 
geographical network responsible for the core policies and procedures governing each individual 
system are nationwide (DeLaurentis, 2007). 

Knowing one measure regarding a system presages a degree of comprehension of an alternate 
measure of the system (Kuhn, 1962). Systems engineering ‘integration’ could be defined as the 
act of ‘combining two different knowledge representations together’ (Kim & Porter, 2007). 
The integrative framework that is systems integration serves as a tool to analyze the anatomical 
basis for an item; the act of integration serving to identify both missing objects, quantities, and 
processes (Langford, 2007). 

The use of academic faculty performance measurement tools has been the subject of the academic 
literature since 1961 (Gustad, 1961). The evaluation ofa researcher’s output is of obvious interest 
due to the benefits of developing criteria that are not biased or unfair in any way. If such a metric 
could be devised, it could serve as the basis for research project funding, faculty comparison, 
promotion, and management of the same (Sidiropoulos, Katsaros, & Manolopoulos, 2007). In an 
attempt to answer some of the drawbacks of use of such simple indices, researchers have developed 
a myriad of academic productivity indices based upon aggregation, time, seniority, dynamic 
properties, multi-dimensional, trend, etc. based metrics in that they combine mathematically 
other simple indexes into new ones (Garcia-Perez, 2009; Boell & Wilson, 2010). 
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In terms of evaluation of the various indices outlined above with respect to quantitatively tying 
results to specific process or outcome factors, there is precious little research, however. An 
example of such quantitative empirical analysis that could be found in the literature has been used 
to identify the “input-output” efficiency of Chinese universities (Chen, Shen, & Fang, 2010). 
The six output indicators for this study were scientific and technical awards value, academic 
writing, academic theses, national research projects, number of patents granted, and tech transfer 
income. These factors were correlated against input factors which included numbers of various 
staff personnel in various categories (by number) (Chen et ah, 2010). 

In the present case, we are looking for quantities to tie and compare directly against individual 
research project system costs; we will postulate the existence of a metric: Faculty Integrated 
Outputs. We will assume that this is composed of the digital sum of all individual research program 
subject area publications, presentations, reports, and citations on the one hand, and student theses 
and dissertation numbers for a particular funded research program within two years of the onset 
of program project funding. The two year figure is based upon the observed local output data. 
An NPS principal investigator is seen to normally publish at least one scholarly article within 
a two year time frame commencing after project onset. Similarly, students are seen to normally 
produce a M.S. degree thesis within two years of project onset. In another research university 
implementation, either of these two values may be different. We are also specifically including two 
separate categories of “FIO” in our discussion in order to recognize the dynamic yet dual nature of 
research administration in the university environment: between the need to publish in whichever 
form to meet the faculty members academic research performance milestones, and the need to 
balance those against the need to meet the teaching institutions requirement for passing along 
information to the next generation of academics/students via theses & dissertations. 

The expression for the FIO variables of “Publications”, and “Dissertation & Thesis” outputs 
generated are clearly terms for “performance” as, in totality or individually, they are commonly 
accepted metrics of functionality (Langford, 2012). The measurement of them are also assumed to 
be “continuous and quantifiable” (Langford, 2012). The individual performance metrics are also 
seen to be impacted by both temporal changes and events. While they are also each individually 
discrete, we also will make the assumption that we may manipulate them mathematically as a 
means of determining a total FIO quotient for each faculty member, department, school, 
Center, or institute. This approach is common in Academia. Several authors, as stated above, 
have attempted to identify various mathematical constructs (ex: h-index, m-index, relative value 
unit, academic readiness level, etc.) (Iyengar, Wang, Chow, & Charney, 2009; Thompson, Callen, 
and Nahata, 2009; Mezrich & Nagy, 2007). None of these expressions, however, is intended 
to constitute a single “determinator” in that one citation, publication, or thesis is any more 
valuable than any other, just that we will make the assumption that they may be manipulated 
mathematically as a means of quantifying and discovering relationships between FIO and any 
other research administrative developed metrics. 

Before displaying several FIO and other critical example FIO data, a status review will be 
conducted with respect to the characteristics of the presumed model that has been proposed. A 
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postulated multiple input (assumed four independent variable-labor, travel, equipment, contract) 
single output(s) (Faculty Integrated Output-Dissertation & Thesis or Publication) model. What 
we do not have at the moment is more than a general theoretical idea of how the independent 
variables should interact in order to produce a predictive trend of how the individual NPS 
researcher and research groups “products” should evolve. The next question is obviously, why the 
concern? There are computationally effective methods available to the research administration 
community that can optimize such “black-box models” with undetermined structurally complex 
interactions. Neural Networks and genetic algorithms are but two possible examples, hi order 
to narrow down the mathematical analysis approach further, we should consider what exactly 
are the attributes of the FIO product that we are looking for? The insight developed from the 
FIO use is meant to motivate deeper involvement in both discovering as well as fixing the issues 
brought to light through their use. Secondly, given the scope, breadth, and multiple interaction 
pathways experienced by all research projects at a university, one would like to believe that the 
widest possible search process done in order to find the best near-optimal solution would seem 
to be appropriate. As the introduction of a new FIO metric is also expected to be continuous, the 
solution space for any research group FIO trend data is expected to evolve over time. For example, 
if the labor system cost solution factor evolves as a more important component of a total FIO data 
for one research group versus another over time, this in itself provides motivation for determining 
the reasons why. How the FIO would also be implemented into the day to day operations of the 
research administrative organization is also important. We are not talking about a requirement to 
generate instant and complex time sensitive solutions. We are talking about the incorporation of 
data over long periods of time wherein the individual “optimal FIO solution” from one research 
entity is combined with other metrics under the watchful eyes of presumably very experienced 
and trained research administrative personnel before any decisions are made. Lastly, as all system 
independent variables are tracked as cost components, which individual system components (and 
combinations thereof) play the largest role in research projects X or research groups Y work 
should be easily available (i.e„ the ability to choose and adapt other functional constraints on 
the functionality of the problem solution should be easily adaptable) for research administration 
observation. 

Research Questions 

In the context of university research administration, the specific research questions are: 

1. May a research institution develop a mathematical computational analysis method linking 
the independent system variables of Labor, Travel, Equipment, and Contracts on the one 
hand, and the dependent variable of FIO on the other? 

2. Provided we are successful in (1.), what increased situational awareness does the analysis 
of the Systems to FIO data provide research administrators? 

Chosen Computational Method 

There are several basic computational analysis options available to analyze the above quantitative 
multi-variate data. The authors have chosen to use the Mahalanobis-Taguchi System (MTS) 
(Taguchi & Jugulum, 2002) analysis technique for this proof-of-concept trial, hi the setting 
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of research administration operations, MTS has been shown to offer a number of important 
advantages over other computational analysis methods: (1) it offers the possibility of independent 
variable reduction; (2) it can identify both “abnormal” and “normal” data sets; (3) the degree 
of “abnormality” can be measured relatively simply; (4) it requires no assumptions as to the 
distribution form for the data; (5) the controls governing what is “normal” and what is “abnormal” 
can be easily changed and the effects viewed; and (6) it is not computationally complex (Kumano, 
Mikami, & Aoyama, 2011; Holcomb, 2016). 

MTS is comprised of four stages. Performing a multivariate analysis utilizing MTS requires 
the obligatory data. Once the data is obtained, including all variables that can influence any 
outcomes, the researcher must determine the parameters of “normal” versus “abnormal” data. 
From these separate identities, a Mahalanobis Space (MS), composed of the differentiated 
Mahalanobis Distances (MD) for each sample, is calculated. The MD represents a scaled distance 
of all variables delineating the correlation between each. Two calculation methods are known 
for determining MD: (l) Inverse Matrix Method; and (2) Gram-Schmidt Orthogonal Process 
(Taguchi & Jugulum 2002). The Inverse Matrix Method (IMM) will be used in this work. The 
IMM is described in Equation 1. 

1 . MDj-Z'ijC 'Z;j 

MDj is a squared distance and Z,y, shown in Equation 1, is the standardized vector, Z’y is 

the transpose of the standardized vector, and C" 1 is the inverse of the correlation matrix 

of the data. 

2. Z ij =(X ij -m i )/s i 

Xijis the data, mj\s the mean of a specific column, and s t is the standard deviation. 

In stage two of MTS, the MDs for the abnormal group are calculated and compared to the normal 
group values. To calculate the abnormal MDs, the standardized vectors must be standardized 
using the mean and standard deviation from the normal group. Also, the correlation matrix 
and inverse correlation matrix come from the normal group as well. The abnormal threshold is 
validated if the MDs for this group have higher values than for the healthy group. 

The third step in MTS is to determine all functional variables that influence the outcome using 
Orthogonal Arrays (OA) and Signal-to-Noise Ratios (SNRs). In MTS, an OA is a survey 
instrument with two identities; either the variable is present or it is not. For each run specified by 
the OA, MDs are recalculated. The SNRs attempts to recognize all functional variables compared 
to variations in the system. SNRs are calculated from the MDs of each run using Equation 3. 

3. s™ r[yE(]4)] 

Non-functional variables are removed based on the overall gain of the system. Only functional 
variables that increase the systems overall gain are retained. The last stage of MTS is the condition 
of monitoring the MD scales while applying either suitable conclusions or remedial actions based 
on the observed values. 
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Ill the present situation research administration case, where there are expected to be a large 
variation in the incidences of correlations because of the diversity of research and principal 
investigators as well as the individual and group small data sets, as long as we select the proper 
“normal” and “abnormal” groups correctly, MTS would seem to offer an initial advantage over 
any other technique because of the prior stated reasons. In this proof-of concept trial, we have 
set the initial publication and thesis & dissertation abnormal/normal differentiation data set 
factor boundary at zero. Thus, if the research unit (e.g., department, Center, or Institute), faculty 
type (e.g., tenure track versus non-tenure track), or individual principal investigator produces 
a publication or thesis/dissertation, it is considered “normal.” If they do not, it is considered 
“abnormal.” In future iterations, where the normal/abnormal data set boundary could be set at 
a non-zero quantity, as in the case of mandated research project quarterly or monthly reports, 
we would have to possibly modify the selection criteria to include cluster analysis or publication 
quantity averaging for determination of the boundary standard. 

System Uncertainty Sources 

In the desire to link research project system costs to research project outputs of D/T numbers and 
Publication output, we have assumed that the expressions for system costs are capturing the true 
costs in total annual labor $, travel $, equipment $, and contract $ as they relate to those same 
outputs. How valid is this ? 

From work in the 1980 s concerning production “technical efficiencies” (Schmidt & Sickles, 1984) 
we know that at least some airline companies’ ability to produce “maximal output” is in the range 
of seventy to one hundred percent. This would also imply that there are inefficiencies on the order 
of zero to thirty percent. Such items as “marketing research costs”, testing costs, and administrative 
costs are all not usually correctly costed in research project accounting (Skaife, Swenson, & 
Wangerin, 2013). There have also been so-called R&D “intangible” expenses identified (Siegel & 
Borgia, 2007) to include HW/SW, technical expertise, training programs, and customer service 
capability that could be left out of any research project total accounting scheme (Johnson, 1964). 
Hence, we know that there are likely to be sufficient research project uncertainty components 
that are missing from individual or group research projects depending on whether or if these 
“costs” are accounted for. In an attempt to quantify the specific NPS case, Langford (2015) has 
calculated there may be a ~20-25% uncertainty rate in the labor cost accounting alone. To deal 
computationally with these cost uncertainty “noise” issues, a fifth random variable has been 
inserted into the MTS calculation along with the other four independent variable system costs. 
Because of the number of variables and the lack of any known variable interactions, the L8(2 7 ) 
Taguchi Table was chosen as the orthogonal array for use in our MTS calculations (Taguchi & 
Jugulum, 2002). 
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Results 

Data Sources 

Between 2013 and 2015, the authors collected and reduced the 2010-2012 NPS FIO research 
output and matched them with the correlated system cost data for over three thousand research 
projects from fourteen different university departments, centers, and institutes in order to 
establish a possible basis for an additional system research administrative information metric tool. 
Dissertation and Thesis (D/T) and Publication (Pubs) specific MTS Gain factors were obtained 
for the fourteen research units as well as the faculty sub-unit type Tenure (TT), Non-Tenure 
Track (NTT), and individual Principal Investigators (Pis) in some cases. 

Data Summary 


Table 1. NPS Group MTS Gain Factor Success Percentages. 


Fiscal Year 

D/T MTS Gain Factor 

Success Percentage 

Pnbs MTS Gain Factor 
Success Percentage 

10 

78.5 

78.5 

11 

92.8 

85.7 

12 

71.4 

57.1 


Table 2. NPS Faculty Type MTS Gain Factor Success Percentages 


Fiscal Year 

D/T MTS Gain Factor 

Pubs MTS Gain Factor 


Success Percentage 

Success Percentage 


(TT/NTT) 

(TT/NTT) 

10 

0 0 

00 

11 

7.1/14.2 

14.2/14.2 

12 

7.1/7.1 

7.1/7.1 
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Table 3. Group 4 Integrated TT/NTT/PI MTS Gain Factors vs. Budget Expenditure Percentages. 


FY 

Labor 

Budget 

% 

Travel 

Budget 

W 

Equqnnent 
Budget % 

Contract Budget % 

10 

.71 

.06 

.04 

.18 

n 

.67 

.06 

.009 

25 

Individual 

E 

35 

.09 

.02 

53 

Individual 

B 

64 

.03 

.001 

33 

12 

.82 

.06 

.018 

.09 

IT 

.87 

.07 

.017 

.05 

NTT 

.81 

.06 

.018 

.10 

FY 

Labor 

Travel 

Equqnnent 

Contract 

D'T 

Gain 

Gain 

Gain 

Gain 

10 

- 

- 

- 

- 

n 

5.2 

4.75 

-2.2 

-1.5 

Individual 

EMIT 

5.37 

11 

69 

321 

12 

3.68 

052 

2.02 

058 

NTT 

6.27 

5.91 

425 

085 

FY Pubs 

Labor 

Gain 

Travel 

Gain 

Equipment 

Gain 

Contracts Gain 

10 

8.48 

4.91 

-1.23 

0.73 

li 

3.62 

424 

-2.4 

063 

Individual 

BTT 

109.9 

41 

52 

43 

12 

1.79 

0.71 

-0.56 

093 

NTT 

4.6 

1.04 

239 

226 


FY 11 D/T Note: FY11 MS Space (MD values: 1.56. 2.36,2.42, 9..6.Q) Reduced 
variable MS space (MD values: (135, 2.43). Correlation Coefficient= .98 
FY10 -12 Pubs Note: FY10 MS space(MD values: 1.19,4.2, .71,3^5). Reduced 
variable MS space (MD values: 131,4.34, 3J9Z)- Conelation Coefficient=.993; 
FY11 MS space (MD values: 2.69, 10.08,1.81, Reduced variable MS space 
(MD values: 1.89, 8.7,1.3)- Conelation Coefficient=.995; FY12 MS space (MD 
values: 6.93, 333, 3.71, TJX)- Reduced variable MS space(MD values: 5.5, 4.1, 
1.9). Conelation Coefficient=.941 
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Table 4. Group 4 Calculated Standard Errors and Signal-to-Noise Ratios for FY10-12 D/T Data. 


Group 4 DT Data Set 

Standard 

Error(FY10,’ll,’12) 

S/N Ratio(FY10,Tl,T2) 

Normal 

(-, ,406,.266) 

(-, 2.45,3.75) 

Abnormal 

rr -1 

O 

GO 

(-,5.54 ,7.58) 

Improvement 

(-, .226,. 135) 

(-, 3.09,3.83) 


Table 5. Group 4 Calculated Standard Errors and Signal-to-Noise Ratios for FY10-12 Pubs Data. 


Group 4 Pubs Data Set 

Standard 

Error(FY10,Tl,T2) 

S/N Ratio(FY10.Tl,T2) 

Normal 

.185, .297,.200 

5.39, 3.35,4.24 

Abnormal 

.123, .119,.235 

8.124,8.37,4.98 

Improvement 

.062, .178,.035 

2.734, 5.02,0.74 


Table 6. Group 4 TT/NTT/Individual Calculated Standard Errors and Signal-to-Noise Ratios 
for FY10-12 DT and Pubs Data. 


Group 4 NTT DT 11 Ind E 

Standard Error 

S/N Ratio 

Normal 

.461 

2.16 

Abnormal 

.0059 

169.18 

Improvement 

.46 

167.02 


Discussion 

Aligning Findings to the Research Questions 

Research Question 1: System to FIO Analysis Method. The use of the Mahalanobis Taguchi 
System (MTS) method allows for determination of the specific system variables and their 
associated degree of contribution to both of the identified Pubs and D/T outputs at the research 
unit levels. As can be seen in Table 1 the majority of NPS research units (Department, Centers, 
and Institutes) covered in the fourteen groups researched provided MTS Gain Factors which 
identify the systems affecting research outputs. At the faculty sub-unit levels, as can be seen in 
Table 2, the percentages are much lower. This result is not surprising. All groups under MTS 
evaluation require both a minimum number of acceptable independent variable value sets as 
well as adherence to compliance rules for determining the validity/usefulness of the derived 
Mahalanobis Distance. In the case of TT and NTT there are simply less data available. For the 
specific situation of individual NPS Pis researched, the situation is even worse. Over a three year 
period, only five individual Pis researched successfully developed MTS Gain Factors for the 
same reasons. The same MTS Gain Factors were also for individual years and hence, were not 
available for tracking over a multi-year period. Table 3 is used to display the integrated combined 
research group, faculty, and principal investigator results for one of the fourteen research groups 
observed. As can be seen, the differences are indicative of factors impacting research group and 
sub-elements of those groups. 
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Between FY10 and FY12, Group 4s total research budget increased 12%. D/T Gain Factor 
information was not available for FY10. For FY10, all abnormal D/T MD values were negative, 
negating any MTS Gain Factor results. Raw thesis and dissertation output numbers more than 
tripled from FY10 to FY12 for Group 4. In FY11, neither the Equipment nor Contract systems 
contributed to the generation of this group’s D/T output as the MTS Gain Factors were both 
negative. Labor was seen as both the largest budget percentage as well as the largest system 
contributor to the generation of D/T output between FY11 and FY12. 

The publication output numbers from this group increased 13% from FY10-FY12. What is 
interesting here is that the MTS Gain Factor numbers associated with the Equipment system for 
this group remained negative for the three year period. This means that the Equipment system 
variable is not seen as contributing to the Pubs outputs. More in-depth analysis of the specific 
dynamics of this group would have to be done to account for this. 

While there were slight differences between TT and NTT budget percentages in FY12, the 
budget was dominated by the Labor system contribution. NTT personnel were responsible 
for thirty three percent of the raw dissertation and thesis output in FY12 and fifty percent of 
the publications raw output. The Labor system was also seen to dominate the contributions to 
NTT D/T and Pubs contribution outputs. Individual E produced twelve percent of the FY11 
group raw dissertation and thesis output. Individual B produced thirty two percent of the FY11 
raw publications output. For both individuals, there are substantial MTS Gain Factor system 
differences between them and the member research group MTS Gain Factors. Unfortunately, 
there was not enough data to support either a direct TT or NTT comparative analysis for these 
years. More in depth analysis of the specific dynamics of this group would have to be done to 
account for this. 

Research Question 2: Research Administration Situational Awareness. By providing additional 
system information insight into the relevant output behaviors for research institutions, we provide 
an additional system contribution perspective valuable to the execution of research administration. 
As an example, if we have advance knowledge of macro research system changes (ex: changes to 
contracting or travel organization/policy execution), we may have advance knowledge of which 
specific research groups outputs may be vulnerable to those systems in the near term. 
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Conclusion 

Summary 

This study addressed the postulated relationship between research system independent variables 
of Labor, Travel, Equipment, and Contract total annual cost and the dependent variables of 
total bi-annual publications/thesis & dissertation outputs. The three-year study data captured 
the identification of which research system variables are responsible for these same outputs by 
magnitude and degree. The findings of the study also help to identify which critical systems are 
responsible for research group outputs independent ofbudget percentage expenditures. At higher 
levels of research group fidelity, the MTS Gain Factor identification was shown to be problematic 
due to a combination of acceptable variable value sets as well as MTS compliance rules. Through 
such concrete research system identification, however, research administrative personnel may 
have the possibility of directly identifying the association between likely effects of reduced or 
affected research systems on the research outputs themselves. 

Future Research 

The authors will continue to develop the system engineering-based FIO construct seen above 
to include possible extension to a non-research university laboratory data set case. The authors 
will also review any potential system structural effects that may have been responsible for the 
NPS results. As MTS has also been used to forecast other physical system multi-variate systems 
with success (Soylemezoglu, Sarangapani, & Saygin, 2011; Hu, Zhang, & Liang, 2013), we 
will specifically include this possibility in our future research administration information 
development construct. 
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